Papers with Model merging
Exploring Model Kinship for Merging Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Model merging has become one of the key technologies for enhancing the capabilities and efficiency of Large Language Models. |
| Approach: | They propose a model merging strategy that incorporates model kinship to improve model performance. |
| Outcome: | The proposed model merging strategy can yield better performance on benchmark datasets. |
STAR: Spectral Truncation and Rescale for Model Merging (2025.naacl-short)
Copied to clipboard
| Challenge: | Model merging is an efficient way of obtaining a multi-task model from several pretrained models without further fine-tuning. |
| Approach: | They propose a model merging technique that aims at mitigating "merging conflicts" by truncating small components in the respective spectral spaces and then an automatic parameter rescaling scheme to retain the nuclear norm of the original matrix. |
| Outcome: | The proposed model outperforms baseline models on flan-T5 by 4.2% and is robust to hyperparamater choice. |
Mergenetic: a Simple Evolutionary Model Merging Library (2025.acl-demo)
Copied to clipboard
| Challenge: | Recent work shows that combining model merging with evolutionary algorithms can boost performance, but there is currently no library for experimenting with different evolutionary algorithms and merging methods. |
| Approach: | They propose an open-source library for evolutionary model merging that enables easy composition of merging methods and evolutionary algorithms while incorporating lightweight fitness estimators to reduce evaluation costs. |
| Outcome: | The proposed library produces competitive results across languages and tasks using modest hardware. |
An Empirical Study of Multimodal Model Merging (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that model merging can generate a multi-task solution without synchronous training. |
| Approach: | They propose to merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient architecture. |
| Outcome: | The proposed model merging outperforms naive models on various tasks with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% ADE20k. |
Practical Guidelines for Model Merging in LLMs Pre-Training (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing studies on model merging have focused on stable learning rate regimes, but its effectiveness during LLM pre-training remains underexplored. |
| Approach: | They systematically investigate model merging across training phases, focusing on the transition from stable to decaying learning rates. |
| Outcome: | The proposed methods improve performance during stable learning rate regimes but diminish under decay, a phe-nomenon that is linked to reduced checkpoint diversity and lower parameter-space variability. |
PRIME: Ultra-Low-Rank Principal–Residual Model Merging (2026.findings-acl)
Copied to clipboard
Seung-Ho Lee, Kyungsu Lee, Bazarvaani Zuchi, Jeongmin Ahn, Insuk Seo, Donghyeon Jeon, Inho Kang, Seung-Hoon Na
| Challenge: | Existing methods for model merging have been limited by task-specific performance and task-related tasks. |
| Approach: | They propose an ultra-low-rank principal-residual model merging framework that decomposes task vector merging into two stages. |
| Outcome: | Experiments on eight natural language processing tasks show that PRIME outperforms existing models while preserving the task-specific capabilities of the original models. |
Superpose Task-specific Features for Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for model merging are limited by resource demands . recent studies validate the linear representation hypothesis . |
| Approach: | They propose a method that superposes task-specific features from individual models into a merged model. |
| Outcome: | The proposed method outperforms existing methods on multiple benchmarks and models. |
Dynamic Fisher-weighted Model Merging via Bayesian Optimization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing merging approaches involve scaling the parameters model-wise or integrating parameter importance parameter-wise. |
| Approach: | They propose a method for merging model-based models at the parameter level without training data or joint training. |
| Outcome: | The proposed model merging framework outperforms baseline models on validation sets. |
Merger-as-a-Stealer: Stealing Targeted PII from Aligned LLMs with Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Model merging is a promising approach for updating large language models . but unmonitored mergers can introduce significant security vulnerabilities . |
| Approach: | They propose a model merging attack surface where a malicious merger can extract PII from an aligned model with model merg. |
| Outcome: | The proposed framework can extract PII from an aligned model with model merging. |
MergePrint: Merge-Resistant Fingerprints for Robust Black-box Ownership Verification of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Model merging introduces a novel risk of unauthorized use of large language models due to the high cost of training. |
| Approach: | They propose a model merging method that embeds robust fingerprints into models . they aim to protect LLMs from misappropriation via model merg and model theft . |
| Outcome: | The proposed method enables black-box ownership verification without accessing model weights or intermediate outputs. |
Revisiting Model Interpolation for Efficient Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models that interpolate weights of two specialized models can be abused for efficient reasoning. |
| Approach: | They propose to merge two specialized models and create a model that combines efficiency and efficiency. |
| Outcome: | The proposed method outperforms existing models on efficiency and effectiveness. |
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters (2026.acl-long)
Copied to clipboard
| Challenge: | Existing research lacks systematic analysis of the applicability and methodology of cross-modal skill injection. |
| Approach: | They investigate the applicability and methodology of cross-modal skill injection by integrating a domain-expert LLM into a VLM. |
| Outcome: | The proposed method enables transfer of domain-specific expertise from Large Language Models (LLMs) to VLMs without incurring additional training data requirements or significant computational overhead. |
Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy (2025.findings-acl)
Copied to clipboard
| Challenge: | Model merging is a widespread technology in large language models that integrates multiple task-specific LLMs into a unified one. |
| Approach: | They propose a model merging approach that trains a phishing model capable of stealing privacy using a privacy phish instruction dataset. |
| Outcome: | The proposed model cloaking method mimics a specialized capability to conceal attack intent, luring users into merging the phishing model. |
3DM: Distill, Dynamic Drop, and Merge for Debiasing Multi-modal Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Multi-modal Language Models have shown remarkable performance in multimodal tasks . however, these models often exhibit inherent biases that compromise their reliability and fairness. |
| Approach: | They propose a framework that integrates Distill, Dynamic Drop, and Merge to address these challenges. |
| Outcome: | The proposed framework outperforms existing methods in balancing debiasing and improving performance on the MMSD2.0 sarcasm detection dataset. |
Will it Merge? On The Causes of Model Mergeability (2026.findings-acl)
Copied to clipboard
| Challenge: | Model merging has emerged as a promising technique for combining fine-tuned models into a single expert model without retraining. |
| Approach: | They propose a model merging technique that preserves weak model knowledge . they define mergeability as a property of model updates that captures how well they retain trained knowledge when merged with other model updates. |
| Outcome: | The proposed method preserves weak knowledge in the base model. |
Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing research on model merging focuses on optimizing model performance and minimizing backdoors. |
| Approach: | They propose a backdoor attack targeting model merging in Large Language Models that creates a unified model for multi-domain tasks. |
| Outcome: | The proposed attack is effective across models, merging algorithms, and tasks while maintaining utility across tasks. |
Personalized Language Models via Privacy-Preserving Evolutionary Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for personalization in language models lack explicit mechanisms for privacy preservation. |
| Approach: | They propose a Privacy-Preserving Model Merging via Evolutionary Algorithms to optimize utility while minimizing privacy risks. |
| Outcome: | The proposed approach outperforms baseline models on the LaMP benchmark and achieves 45% improvement in task performance. |
Enhancing Multilingual Reasoning via Steerable Model Merging (2026.findings-acl)
Copied to clipboard
Zhuoran Li, Rui Xu, Jian Yang, Junnan Liu, Zhijun Chen, Qianren Mao, Hongcheng Guo, Jiaheng Liu, Likang Xiao, Ming LI, Xiaojie Wang
| Challenge: | Model merging is an effective technique for composing the capabilities of a multilingual model and a reasoning model. |
| Approach: | They propose a model merging framework that modulates the contribution of each source model. |
| Outcome: | Experiments show that the proposed model merging framework outperforms strong baselines on multilingual reasoning benchmarks across 21 different languages. |
Why Do More Experts Fail? A Theoretical Analysis of Model Merging (2026.acl-long)
Copied to clipboard
Zijing Wang, Xingle Xu, YongKang Liu, Yiqun Zhang, Peiqin Lin, Shi Feng, Daling Wang, Xiaocui Yang, Hinrich Schuetze
| Challenge: | Existing methods for model merging struggle to maintain performance gains as the number of merged models increases. |
| Approach: | They propose a Reparameterized Heavy-Tailed method to extend the merged model’s coverage and enhance performance. |
| Outcome: | The proposed method extends the merged model’s coverage and enhances performance on 19 benchmarks, including knowledge-intensive and general-purpose tasks. |